REPORT  DOCUMENTATION  PAGE 


Form  Approved  OMB  NO.  0704-0188 


The  public  reporting  burden  for  this  coilection  of  information  is  estimated  to  average  1  hour  per  response,  inciuding  the  time  for  reviewing  instructions, 
searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  compieting  and  reviewing  the  coiiection  of  information.  Send  comments 
regarding  this  burden  estimate  or  any  other  aspect  of  this  coilection  of  information,  including  suggesstions  for  reducing  this  burden,  to  Washington 
Headquarters  Services,  Directorate  for  information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Ariington  VA,  22202-4302. 
Respondents  shouid  be  aware  that  notwithstanding  any  other  provision  of  iaw,  no  person  shaii  be  subject  to  any  oenaity  for  failing  to  comply  with  a  coiiection 
of  information  if  it  does  not  dispiay  a  currentiy  vaiid  OMB  controi  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1 .  REPORT  DATE  (DD-MM-YYYY)  2.  REPORT  TYPE 

New  Reprint 


4.  TITLE  AND  SUBTITLE 

Sensitivity  Analysis  of  Compressive  Sensing  Solutions 


6.  AUTHORS 
Liyi  Dai 


3.  DATES  COVERED  (From  -  To) 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 
W911NF-04-D-0003 


5c.  PROGRAM  ELEMENT  NUMBER 
611102 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAMES  AND  ADDRESSES 

North  Carolina  State  University 

2701  Sullivan  Drive 

Admin  Services  III;  Box  7514 

Raleigh,  NC  27695  -7514 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS 
(ES) 

U.S.  Army  Research  Office 
P.O.Box  12211 

Research  Triangle  Park,  NC  27709-2211 


8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 
ARO 


11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

62483-CS-SR.lO 


12.  DISTRIBUTION  AVAILIBILITY  STATEMENT 
Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 

The  views,  opinions  and/or  findings  contained  in  this  report  are  those  of  the  author(s)  and  should  not  contrued  as  an  official  Department 
of  the  Army  position,  policy  or  decision,  unless  so  designated  by  other  documentation. 


14.  ABSTRACT 

The  compressive  sensing  framework  finds  a  wide  range  of  applications  in  signal  processing  and  analysis.  Within 
this  framework,  various  methods  have  been  proposed  to  find  a  sparse  solution  x  from  a  linear  measurement  model  y 
=  Ax.  In  practice,  the  linear  model  is  often  an  approximation.  One  basic  issue  is  the  robustness  of  the  solution  in  the 
presence  of  uncertainties.  In  this  paper,  we  are  interested  in  compressive  sensing  solutions  under  a  general  form  of 
measurement  y  =  (A  +  B)x  +  v  in  which  B  and  v  describe  modeling  and  measurement  inaccuracies,  respectively. 

Ty-v  y-v  1  T  rr^  y-v  y-v  y-i  y-v^y^  y-i  .«  4- .«  -r  t  .«  t  y-v  4^  y,  yv  1 1  1  yv  y-i  4-  yv  -t  4^  ^  4-  ^  yv  1  yv  yl  y-v  1  y»  y-v*«*«yv  yv  y-v  y*  y-i-i  i  *«y-v»yv%  y-v»yfc  4-  -t  y*yfcyfc-ii*«y*yfcTr  -rr  Ly'  r  yv  yfc4- 


15.  SUBJECT  TERMS 

compressive  sensing,  sparse  solutions,  sensitivity  analysis,  robustness 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

15.  NUMBER 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

ABSTRACT 

OF  PAGES 

UU 

UU 

UU 

UU 

Liyi  Dai 


19b.  TELEPHONE  NUMBER 
919-549-4350 


Standard  Form  298  (Rev  8/98) 
Prescribed  by  ANSI  Std.  Z39. 18 


Report  Title 

Sensitivity  Analysis  of  Compressive  Sensing  Solutions 

ABSTRACT 

The  compressive  sensing  framework  finds  a  wide  range  of  applications  in  signal  processing  and  analysis.  Within  this 
framework,  various  methods  have  been  proposed  to  find  a  sparse  solution  x  from  a  linear  measurement  model  y  = 
Ax.  In  practice,  the  linear  model  is  often  an  approximation.  One  basic  issue  is  the  robustness  of  the  solution  in  the 
presence  of  uncertainties.  In  this  paper,  we  are  interested  in  compressive  sensing  solutions  under  a  general  form  of 
measurement  y  =  (A  +  B)x  +  v  in  which  B  and  v  describe  modeling  and  measurement  inaccuracies,  respectively.  We 
analyze  the  sensitivity  of  solutions  to  infinitesimal  modeling  error  B  or  measurement  inaccuracy  v.  Exact  solutions 
are  obtained.  Spec  fically,  the  existence  of  sensitivity  is  established  and  the  equation  governing  the  sensitivity  is 
obtained.  Worst-case  sensitivity  bounds  are  derived.  The  bounds  indicate  that  sensitivity  is  linear  to  measurement 
inaccuracy  due  to  the  linearity  of  the  measurement  model,  and  roughly  proportional  to  the  solution  for  modeling 
error.  An  approach  to  sensitivity  reduction  is  subsequently  proposed. 


REPORT  DOCUMENTATION  PAGE  (SF298) 
(Continuation  Sheet) 

Continuation  for  Block  13 


ARC  Report  Number  62483. 10-CS-SR 
Sensitivity  Analysis  of  Compressive  Sensing  So... 


Block  13:  Supplementary  Note 

©  2015  .  Published  in  Frontiers  in  Robotics  and  Al:  Sensor  Fusion  and  Machine  Perception,  Vol.  Ed.  0  2,  (1)  (2015),  (,  (1).  DoD 
Components  reserve  a  royalty-free,  nonexclusive  and  irrevocable  right  to  reproduce,  publish,  or  otherwise  use  the  work  for 
Federal  purposes,  and  to  authroize  others  to  do  so  (DODGARS  §32.36).  The  views,  opinions  and/or  findings  contained  in  this 
report  are  those  of  the  author(s)  and  should  not  be  construed  as  an  official  Department  of  the  Army  position,  policy  or  decision, 
unless  so  designated  by  other  documentation. 

Approved  for  public  release;  distribution  is  unlimited. 


Fifty-second  Annual  Allerton  Conference 
Allerton  House,  UlUC,  Illinois,  USA 
October  1-3,  2014 

Sensitivity  Analysis  in  RIPless  Compressed  Sensing 


Abdolreza  Abdolhosseini  Moghadam,  Mohammad  Aghagolzadeh,  Hayder  Radha 


Abstract —  Sensitivity  analysis  in  optimization  theory  explores 
how  the  solution  to  a  particular  optimization  problem  changes 
as  the  objective  function  or  constraints  of  such  optimization 
problem  perturb.  A  recent  and  yet  important  class  of  opti¬ 
mization  problems  is  the  framework  of  compressed  sensing 
where  the  objective  is  to  find  the  sparsest  solution  to  an  under¬ 
determined  and  possibly  noisy  system  of  linear  equations.  In 
this  paper,  we  show  that  by  utilizing  some  tools  in  sensitivity 
analysis,  namely  Invariant  Support  Sets  (ISS),  one  can  Improve 
certain  developed  results  in  the  field  of  compressed  sensing. 
More  specifically,  we  show  that  in  a  noiseless  and  RIP-less 
setting  [II],  the  recovery  process  of  a  compressed  sensing 
framework  is  a  binary  event  in  the  sense  that  either  all  vectors 
with  the  same  support  and  sign  pattern  can  be  recovered  from 
their  compressive  samples  or  none  can  be  estimated  correctly. 
However,  in  a  noisy  and  RIP-less  setting,  recovering  only  one 
signal  from  its  limited  noisy  samples  guarantees  that  there  exist 
signals  (possibly  even  with  different  supports  and  sign  patterns) 
and  noise  vectors  that  shall  be  recovered  with  good  accuracies 
by  using  Lasso. 

Index  Terms — sensitivity  analysis,  compressed  sensing, 
compressive  sampling.  Invariant  Support  Set. 

1.  INTRODUCTION 

An  important  optimization  problem  which  appears  in  a 
wide  range  of  applications  [1],  [2],  [3],  [4],  [5]  is  the  problem 
of  compressed  sensing  [6],  [7],  [8],  [9],  [10],  [11],  where  the 
objective  is  to  recover  the  sparsest  (i.e.  mostly  zero)  vector 
X  €  K"  from  limited  linear  samples  y  =  Ax  where  A  € 
and  by  limited  we  mean  that  the  number  of  samples 
is  less  than  the  size  of  the  vector  x,  i.e.  m  <  n.  On  the  other 
hand,  sensitivity  analysis  [13],  [14],  [16]  in  optimization 
problems  and  systems  of  equations  in  general,  investigates 
how  the  solution  to  an  optimization  problem  or  that  system 
of  equation  changes  by  perturbation  in  constraints  or  the 
objective  function.  Despite  the  importance  of  such  notion 
in  the  post  calculation  and  analytical  phase,  to  the  best  of 
our  knowledge,  no  attempts  has  been  made  so  far  to  deploy 
that  kind  of  analysis  in  the  compressed  sensing  framework. 
In  this  paper,  we  show  that  certain  sensitivity  analysis  tools 
improve  some  solid  results  already  developed  in  the  field  of 
compressed  sensing. 

It  is  well  known  that  the  success  of  finding  the  sparsest 
solution  that  satisfies  the  system  of  linear  equation  y  =  Ax 
heavily  depends  on  the  quality  of  the  sensing  matrix  A.  Early 
papers  in  the  field  of  compressed  sensing  [6],  [7]  showed  that 
if  the  matrix  A  has  a  property  called  Restricted  Isometry 
(RIP),  then  the  sparsest  solution  to  the  noiseless  system  of 
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y  =  Ax  has  the  minimum  £i  norm,  and  hence  the  sparsest 
vector  could  be  found  by  linear  programming.  In  a  noisy 
setting,  where  equations  are  contaminated  by  the  noise  vector 
e  (i.e.  y  =  Ax  +  e),  a  popular  technique  (which  provably 
works)  to  recover  x  from  y  is  unconstrained  Lasso  [12], 
which  can  be  cast  as  a  quadratic  programming  optimization 
problem. 

After  the  introduction  of  RIP,  other  sufficient  conditions 
to  guarantee  successful  recovery  were  proposed  for  the 
matrix  A  (for  instance  Null  Space  Property  (NSP)  [17]). 
However,  proving  or  verifying  that  a  matrix  has  RIP  or 
NSP  with  the  minimal  number  of  rows  (to)  turns  out  to 
be  quite  challenging  or  even  impossible  except  for  a  few 
ensembles  of  well-known  matrices  such  as  matrices  with 
entries  following  a  Gaussian  distribution.  In  the  more  recent 
work  of  [11],  authors  showed  that  as  long  as  the  sensing 
matrix  A  has  isotropy  and  incoherence  properties  (see  [11] 
for  definitions),  then  the  matrix  A  should  be  a  “good”  sensing 
matrix  candidate  for  the  compressed  sensing  framework. 
What  signifies  that  work  is  the  fact  that,  those  conditions  are 
much  weaker  than  RIP  or  NSP  and  more  importantly  much 
easier  to  verify.  Clearly  achieving  such  strong  result  in  [11] 
should  not  be  free  and  indeed,  the  universality  of  matrices 
with  RIP  is  given  up  in  that  work'.  More  specifically,  that 
work  shows  that  the  probability  of  recovering  a  fixed  but 
arbitrary  signal  x  from  compressive  samples  y  =  Ax  is  high 
(for  instance  at  least  1  —  0(1) /n  where  n  is  the  length  of 
the  unknown  vector  x),  but  such  matrix  may  not  recover 
all  imaginable  sparse  vectors.  In  this  paper,  by  using  simple 
arguments,  we  show  that  isotropy  and  incoherence  properties 
have  even  greater  consequences  and  in  fact,  recovering  only 
a  fixed  but  arbitrary  signal  from  its  compressive  samples 
deterministically  guarantees  that  there  are  possibly  infinitely 
many  other  signals,  sometimes  even  with  different  supports 
(from  the  recovered  one)  which  could  be  recovered  from 
their  compressive  samples  as  well  and  still  the  quality  of 
recovery  will  be  high.  To  that  end,  we  use  the  notion  of 
“Invariant  Support  Set”  (ISS)  [13],  [14].  Broadly  speaking, 
ISS  studies  under  which  perturbations  in  the  constraints 
of  an  optimization  problem,  the  support  of  the  solution 
to  that  perturbed  optimization  problem  stays  unchanged. 
In  a  noiseless  scenario,  we  show  that,  if  the  compressed 
sensing  framework  can  exactly  recover  a  signal  x  from  pure 
compressive  samples  y  =  Ax  then  any  other  signal  u  with 
the  same  support  and  sign  pattern  of  x,  could  be  recovered 
exactly  from  q  =  Au.  This  extends  the  applicability  of 

'Universality  of  a  pair  of  sensing  matrix  and  a  decoder  means  that,  the 
pair  of  decoder  and  the  fixed  sensing  matrix  could  recover  any  sufficiently 
sparse  vector  from  its  compressive  samples. 
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[11]  from  “a  fixed  but  arbitrary  signal”  to  “all  signals  with 
the  same  support  and  sign  pattern”.  In  the  noisy  scenario, 
we  show  that  if  the  compressed  sensing  framework  has 
successfully  recovered  a  signal  x  from  noisy  compressive 
samples  y  =  Ax+e  and  if  such  recovery  was  not  by  chance^, 
then  there  exist  infinitely  many  other  signals  (even  with 
supports  different  from  x)  and  noise  vectors,  for  instance  u 
and  ^  respectively  for  which.  Lasso  [12]  returns  an  accurate 
estimate  of  u  from  noisy  samples  q  =  Au  + 

This  paper  is  organized  as  follows:  in  Section  II,  we  show 
how  ISS  might  be  utilized  for  the  sensitivity  analysis  of  the 
compressed  sensing  framework.  Section  III  presents  some 
numerical  simulations  and  Section  IV  concludes  the  paper. 

II.  SENSITIVITY  ANALYSIS  IN  COMPRESSED 
SENSING 

Before  proceeding  to  the  main  results,  we  introduce  some 
notations  used  in  this  paper.  Eor  a  natural  number  m,  we 
define  the  set  [to]  :=  {1,2,...,  to}.  Eor  sub-matrix  selection, 
we  adopt  Matlab  notations.  Eor  instance,  let  A  G  be  an 

mxn  real-valued  matrix,  p  C  [to]  and  q  C  [nj.  Then  by  A.  p 
(and  Ag^.  respectively)  we  mean  sub-matrix  of  A  which  is 
formed  by  selecting  columns  (and  rows)  indexed  by  those  in 
p  (and  q).  Cardinality  of  a  set  I  is  denoted  by  Card{l}.  The 
support  of  a  vector  t,  denoted  by  Support{t}  is  the  set  of 
indices  where  t  is  non-zero,  i.e.  Support{t}  :=  {i  :  ti  ^  0}. 
By  we  mean  the  complement  of  the  set  1.  Eor  simplicity  of 
notation  and  without  loss  of  generality,  in  this  paper  we  have 
assumed  that  the  unknown  vector  of  interest  to  be  recovered 
is  non-negative  (xi,Ui  >  0,  Vt  G  [n]).  The  extension  to 
the  general  case,  where  signals  can  take  negative  values  is 
straightforward. 

We  build  our  contribution  on  top  of  the  RIPless  theory 
of  [11]  where  we  assume  that  entries  of  the  sensing  matrix 
A  G  are  i.i.d  from  a  distribution  F  with  the  following 

two  conditions: 

1)  Isotropy  condition:  The  distribution  F  is  zero-mean 
with  variance  1/to. 

2)  Incoherence  property:  If  a  ~  F  then  either  determin¬ 
istically  or  with  high  probability  |a|^  <  /i(F)  where 
/i(F)  is  the  “coherence  parameter”  of  the  distribution 
of  F.  The  smaller  this  value  is,  the  fewer  number 
of  samples  (to)  would  be  required  for  the  recovery 
process.  As  in  [11],  we  assume  that  the  matrix  A  has 
p(F)  =  0{l/^/m). 

In  the  following  two  subsections,  we  discuss  consequences 
of  sensitivity  analysis  and  existence  of  invariant  support  sets 
in  the  noiseless  and  noisy  recovery  cases,  respectively. 


a;  =  argmin  1^0  s.t.  y  =  Az,zF0  (1) 

Z 

Here,  1^  is  an  all-one  row  vector  with  size  n  and  z  F  0 
implies  that  all  entries  of  z  are  non-negative. 

Let  LP{A,y)  be  the  solution  to  the  linear  programming 
optimization  problem  of  (1),  i.e.  LP{A,y)  is  a  vector  with 
the  minimal  £i  norm  that  satisfies  y  =  Ax.  The  ISS  interval 
of  LP{A,y)  is  the  set  of  all  vectors  (Ay  G  M™}  such  that 
Support{LP{A,  y)}  =  Support{LP{A,  y  +  Ay)}. 

The  following  Theorem  proves  that  given  a  fixed  sensing 
matrix  A  and  in  a  noiseless  setting,  the  recovery  process  of 
all  vectors  with  the  same  support  and  sign  pattern  is  a  binary 
event  in  the  sense  that  either  all  or  none  of  signals  with  the 
same  sign  pattern  and  support  could  be  recovered  from  their 
compressive  samples.  In  other  words,  the  recovery  process 
in  the  noiseless  setting  is  independent  of  the  values  of  the 
signal,  and  instead  depends  on  the  support  and  sign  pattern 
of  that  signal  which  sounds  reasonable. 

Theorem  1.  Assume  that  the  matrix  A  has  weak-RIP 
property  in  [11]  (or  estimates  El  —  E4  in  the  same  paper) 
and  there  exist  a  vector  x  with  Support{x}  =  T  such  that 
LP{A,  y  =  Ax)  =  X.  Then,  for  any  other  vector  u  with 
the  same  support  Support{u}  =  T  and  sign  pattern,  we 
would  have: 

LP{A,  q  =  Au)  =  u 

Proof.  Let  x  =  LP{A,  y).  Then  by  assumptions  of  the  theo¬ 
rem,  X  =  X  and  of  course  Support{x}  =  Support{x}  =  T. 
If  the  perturbation  vector  of  Ay  =  Au  —  Ax  belongs  to 
the  ISS  interval  of  LP{A,y)  then  it  means  that  if  we  solve 
the  perturbed  problem  of  ft  =  LP{A,  y  -f  Ay)  =  LP{A,  q) 
then  Support{u}  =  Support{x}  =  T  hy  the  definition 
of  ISS.  In  words,  in  this  case,  the  support  of  the  vector  is 
estimated  exactly  correct  (note  that  Support{u}  =  T  as 
well).  If  that  happens,  since  A  is  non-singular  on  column 
indices  T  (in  fact,  assumptions  of  El  —  EA  in  [11]  imply 
that  A,  p  is  near  isometry  in  the  noiseless  case),  thus  u  must 
be  u.  To  that  end,  by  theorem  2.5  in  [13]  we  need  to  find 
a  non-negative  vector  r  such  that  A,  =  y  +  Ay  =  Au. 
Clearly  that  vector  exist  which  is  simply  r  =  u,  since  by  the 
assumption  Support{u}  =  T  and  uFQ.  Now,  in  a  general 
case  when  signals  can  take  negative  values,  the  compressed 
sensing  formulation  for  the  recovery  process  is 

a:  =  argmin  ||z||i  s.t.  y  =  Az 

Z 

However,  one  can  re-format  the  above  equation  into  the 
canonical  form  of 


A.  Noiseless  setting 

Assume  that,  A  G  and  a;  is  a  non-negative  fc-sparse 

vector  (j|a:jjo  =  k)  of  length  n  where  fc  <  to  <  n.  In  the 
noiseless  case,  a  compressed  sensing  framework  estimates 
the  original  vector  x  from  samples  y  =  Aa;  as  x  where 

^In  the  sense  that,  A  was  ill-conditioned  but  ||Aa::||  ||a;||  for  that 

particular  x. 


Xp,Xn  =  arg  min  1^ 

z„,z„ 


Zn  ]•  “ 

and  form  the  final  estimate  as  a;  =  Xp  —  a;„.  It  should  be 
clear  that  in  this  general  case,  we  are  still  working  with 
non-negative  vectors  Xp  and  Xn  and  consequently,  our  ISS 


s.t.  y  =  A  —A 
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arguments  for  the  formulation  of  (1)  will  hold  in  this  case 
as  well.  ■ 

Note  that  Theorem  1  is  more  general  than  the  notion 
of  “a  fixed  but  arbitrary  signal”  in  [11]  which  derives  the 
probability  of  recovery  for  only  one  fixed  signal  x.  Although, 
it  can  be  argued  that  all  signals  with  the  same  support  and 
sign  pattern  share  the  same  exact  and  inexact  dual  certificates 
(see  [11]  for  definitions)  and  using  that  fact,  one  can  derive 
a  bound  on  the  error  of  the  estimate  ||u  —  m||.  Nevertheless 
and  as  shown  above,  our  proof  is  much  simpler  due  to  the 
sensitivity  analysis. 

In  the  following,  we  prove  that  if  the  recovery  of  x  from 
noisy  samples  y  =  Ax  +  e  had  been  successful  (in  the  sense 
that  the  estimate  x  has  the  property  that  ||x  -  x\\  =  0{e)), 
then  either  deterministically  or  at  least  with  much  higher 
probabilities  than  [11],  infinitely  many  other  signals  (which 
possibly  could  have  supports  different  from  x)  could  be 
accurately  recovered  from  their  limited  compressive  samples 
as  well. 

B.  Noisy  setting 

For  the  noisy  case  and  also  when  the  vectors  to  be 
recovered  are  compressible  rather  than  exactly  sparse,  an 
approach  to  recover  a  signal  x  from  noisy  measurements 
y  =  Ax  +  e  is  utilizing  the  unconstrained  Lasso  [12] 
formulation: 

i  =  argmin^||?/-A&||2  +  A|j5||i  (2) 

b  L 

Here,  A  is  a  fixed  regularization  parameter  that  can  balance 
the  sparsity  of  the  estimation  ||a:||o  against  the  discrepancy 
of  the  estimate  y  —  Ax.  Again  let  us  consider  the  case 
when  the  unknown  vector  x  is  assumed  to  be  non-negative 
and  exactly  sparse.  Let  QP{A,  y,  A)  be  the  solution  to  the 
following  equivalent  but  canonical  unconstrained  Quadratic 
Programming  (QP)  problem  of  (2): 

a;  =  argminC'’^&  H — h'"  Qb  s.t.  bPO  (3) 

b  2 

Where 

C  =  -A^y  +  Xl,Q  =  A'^A  (4) 

The  dual  problem  of  above  is  in  the  form  of 

It 

x,s  =  argmax  — -5  Q6 

b,s  2 

s.t.  s  —  Qb  =  C,  b,s  P  0  (5) 

In  summary,  to  recover  x  from  noisy  samples  y  =  Ax  +  e, 
one  declares  x  =  QP{A,y,  X)  as  an  estimate  of  x  given 
a  fixed  regularization  parameter  A.  Let  us  use  p  to  de¬ 
note  the  support  of  the  estimate  x  (p  =  Support{x}  = 
Support{QP{A,y,  X)}).  Note  that,  since  this  time  we  are 
working  in  the  noisy  realm,  the  event  that  the  support  of  the 
estimate  is  exact  (i.e.  the  event  of  Support{x}  =  p)  is  very 
unlikely  and  in  practice  p  is  much  larger  than  the  support 
of  the  sparse  vector  x.  In  fact,  in  the  noisy  setting,  p  is  not 
necessarily  a  subset  of  Support{x} .  However,  the  converse 
{Support{x}  C  p)  usually  holds  if  the  recovery  process  has 


been  successful.  Here,  we  assume  that  this  case  happens 
in  the  recovery  of  x  (see  [19],  [20]  for  some  conditions 
on  the  support  recovery).  Now,  let  us  find  ISS  interval  of 
QP{A,y,  X)  which  is  the  set: 

fAy:  1 

\Support{QP{A,y,  X)}  =  Support{QP{A,y  +  Ay,  X)} ) 

Note  that,  this  set  is  always  non-empty  as  the  trivial  zero 
solution  for  Ay  belongs  to  this  set.  Clearly,  we  are  interested 
in  the  case  when  Ay  is  non-zero.  We  show  that  this  would 
be  the  case  and  more  importantly,  if  the  recovery  of  x  had 
been  successful  in  the  sense  that 

||a:-gP(A,y,A)||  =0(||e||) 

then,  corresponding  problems  in  the  ISS  interval  of 
QP{A,y,  X)  could  be  solved  with  good  precision  as  well 
in  the  same  sense.  In  contrast  to  the  noiseless  setting  and  as 
stated  before,  the  recovery  of  a  vector  x  from  noisy  samples 
y  =  Ax  -f  e  could  guarantee  the  recovery  of  signals  with 
supports  possibly  different  from  Support{x}.  The  benefit  of 
using  such  sensitivity  analysis  is  that  otherwise  for  each  new 
support  and  sign  pattern  one  must  find  exact  or  inexact  dual 
certificates  [11]  which  exist  with  a  probability  of  (at  least) 
1  —  0(l)/n.  Although  such  probability  is  high,  it  might  not 
be  sufficiently  high  or  comparable  with  our  result.  This  can 
be  shown  by  noting  that  Card{Support{x}}  could  be  as 
high  as  m  and  there  are  (™)  possible  cases  just  for  supports 
of  size  k  within  Support{x},  and  hence,  the  probability  that 
dual  certificates  for  all  these  possible  supports  exist  could 
be  very  small  at  the  end  (if  the  probability  of  1  —  0(l)/n 
is  used  without  any  improvement).  In  contrast,  steps  in  our 
approach  mostly  hold  either  deterministically  or  at  least  with 
much  higher  probabilities.  And  finally,  our  error  bound  could 
have  smaller  constants  when  compared  to  the  original  work 
in  [11]  when  proper  regularization  parameters  are  set  and  the 
matrix  A  is  well  conditioned. 

To  utilize  results  from  Invariant  Support  Sets,  we  need 
the  following  lemma  which  states  Qpp  :=  A^pA,^p  has  an 
inverse  for  p  =  Support{QP{A,  y,  A)}. 

Lemma  2.  Let  x  be  the  unique  minimizer  to  QP{A,y,  X) 
and  define  p  =  Support{x}.  Then  Card{p}  <  m  and  A,^p 

is  full  rank. 

Proof.  For  now  assume  that  Card{p}  >  m  or  A,^p  is  not 
full-rank.  In  both  cases,  the  null-space  of  A.  p  is  not  empty 
and  there  exists  a  vector  h  C  p  such  that  A,^ph  =  0.  Define 

t  :=  mm  |  —  | 

i^Support{h}  hi 

Now  consider  two  vectors  of  x  +  th  and  x  —  th.  Let  I  = 
{i  :  hi  >  0}  he  indices  where  h  is  positive  and  define  H  := 
t\h\  (i.e.  Hi  =  thi  for  i  &  I  and  Hi  =  —thi  when  i  € 
Support{h} /T).  Note  that  all  entries  of  Xp  are  always  non¬ 
negative  and  Xi  >  Hi.  Therefore,  \xi  +  hi\  =  Xi  +  Hi  when 
i  e  I  and  \xi  +  hi\  =  Xi  —  Hi  in  i  G  P  =  Support{h} /I. 
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Since  x  was  the  unique  solution  to  the  optimization  problem 
of  QP{A,y,  X),  we  have 

^\\y  -  Ax\\l  +  X\\x\\i  <  ^\\y-A{x  +  th)\\l  +  X\\x  +  th\\i 

and  also 

\\\y  -  Ax\\l  +  Aplli  <\\\y-  -  th)\\l  +  A||x  -  th\\i 

However,  since  h  is  in  the  null-space  of  A,  we  have  A{x  + 
th)  =  A{x  —  th)  =  Ax.  This  means  that  |ji:||i  <  ||x  +  th\\i 
and  ||x||i  <  llx  — simultaneously.  Assuming  ||i:||i  < 
II a;  +  th\\i  means 

'^Xi  <  '^{xi  +  Hi)  +  '^{xi  -  Hi) 
iGp  iGl 

and  hence  ||i//||i  >  ||i?ic||i.  Similarly  ||a;||i  <  ||a:  — fft-||i 
means  that 

'^Xi  <  +  Hi)  +  '^{xi  -  Hi) 

iGp  iGl 

This  translates  to  |lH/<:||i  >  |lf?i||i  which  contradicts 
||a;||i  <  ||a:  +  th\\i.  This  means  that,  if  Card{p}  >  m 
or  A:  p  is  not  full-rank  then  one  can  find  a  vector  z  (in 
here  either  th  or  —th)  in  the  null-space  of  A-^p  such  that 
||x-|-z||i  <  ||a;||i,  attaining  a  lower  objective  function 
of  Ally  —  A{x  -\-  z)||2  -I-  A||x  -I-  z||i  which  contradicts  the 
assumption  that  x  is  the  unique  minimizer  to  QP{A,y,  X). 
Thus  both  Card{p}  <  m  and  A.^^p  is  full -rank.  ■ 

In  the  following  theorem,  we  derive  characteristics  of 
signals  u  and  noise  vectors  ^  that  lead  to  perturbations 
Ay  =  A{u  —  x)  +  {^  —  e)  residing  in  the  ISS  of  QP{A,  y  = 
Ax  +  e,  A). 

Theorem  3.  Assume  that  the  matrix  A  has  weak-RIP 
property  in  [11]  (or  estimates  El  —  Ed  in  the  same  paper). 
Furthermore,  assume  that  x,  the  unique  minimizer  to 

QP(A,y  =  Ax  +  e,  A)  satisfies  ||a;  —  x\\  =  0(||e||).  Then 
ISS  of  QP{A,y,  X)  is  not  empty  if  ||A:  p||  is  sufficiently 
small  for  p  :=  Support{x}. 

Proof.  By  Lasso  formulation  and  our  assumptions  Vi  G  p  : 
Xi  >  0.  We  need  to  find  for  which  signal  u  and  noise  vector 
Support{u  :=  QP{A,  q  =  Au  +  A)}  would  stay  at  p. 

To  that  end,  let  us  define: 


Cix} 

=  -A^y  +  Al 

(j{u} 

=  -A^q  +  XI 

Ax 

=  u  —  x 

Ae 

=  ^-x 

AC 

—  (jlA  _  (jlA  — 

z 

=  {l,2,...,n}/p 

Qpz 

N 

II 

Qpp 

II 

-A'^iAAx  +  Ae) 


In  summary,  if  Ay  :=  q  —  y  =  AAx  -I-  Ae  belongs  to 
the  ISS  of  QP{A,y,  X),  then  Support{QP{A,q  =  y  + 
Ay,  A)}  =  p  by  definition.  As  shown  in  [14],  to  find  such 
ISS  interval,  it  is  required  to  find  non-negative  vectors  u  and 
which  satisfy  the  following  two  equations 

-Qpphp  =  =  Cf  >  +  ACp 

+  (6) 

Note  that,  since  x  =  QP{A,y,  X),  there  exists  a  non¬ 
negative  vector  with  the  support  on  z  =  [n\/p  =  p‘^ 
such  that 


-QppXp  =  4"^  =  Qpz£p  + 

Since  A.  p  is  full  rank  (see  Lemma  2),  Qpp  =  A^^A.  p  has 
an  inverse.  Consider  the  vector  u  which  at  indices  p  equals 
to 

Up  = 

and  is  zero  in  other  indices  (i.e.  z  =  \n]/p).  As  [16],  let  us 
use  to  denote  the  Lth  row  of  —Qfp  and  note  that 

Vi  G  p  :  Xi  =  >  0 

Similarly,  for  all  i  €  p 


u,  =  +  ACp)  =x,  +  v^^'iACp  (7) 

Using  full-rank  property  of  Qpp  =  A^^A-  p,  one  can 
represent  any  vector  in  in  terms  of  columns  of 

Qpp.  Let  us  define 

Ae  :=  -Q-^A^pAe  =  -A+pAe,  Ax  :=  -Q-^^Ax 

where  by  A+p  we  mean  the  pseudo-inverse  of  A.  p.  Clearly 
l|Ae||  <  ||A+p||||Ae||  and  Ax||  <  ||Q-pi|| || Aa;||.  Since  the 
recovery  of  x  has  been  successful  and  the  support  of  the 
estimate  x  was  p,  hence  it  is  very  unlikely  that  ||A:  p|| 
(and  hence  IIQj^ll  high-  In  f^ct,  if 

Card{p}  —  fc  is  not  too  big,  then  A,  p  has  good  condition 
number  due  to  weak-RIP.  Nevertheless,  plugging  these  values 
into  (7)  yields: 


iii  =  Xi  +  Axi  -f  Aci  (8) 

For  now  let  us  pause  in  here  and  focus  on  si^\  which  is 
by  definition: 

4“}  =  + 

=  Qp.  -  Qpp  ACp)  +  Ci"}  +  AC, 

=  4"}-gJ,Q;piACp  +  AC, 

Using  the  same  arguments  for  Ax,  one  can  define  AC  := 
Qpp  AC  where  clearly  ||AC||  <  ||gpp^||  ||  AC||.  Therefore 

4“>  A4"^-(l|Ac,iu  +  iig4ACp|ioo)i 
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Where  by  we  mean  entry-wise  greater  than  operator. 
By  off  support  incoherency  (see  Lemma  2.4  in  [11]),  it  is 
straightforward  to  verify  that; 

Pf  {\\Qlz^Cp\\^  >  f||ACp||2)  < 


for 


/ilogn 


As  in  [11],  here  we  have  assumed  /i  (the  coherency 
parameter  of  distribution  of  the  entries  of  A)  is  0{l/y/rn). 
Let  us  assume  that  <  \  =  0{k).  Then  by  properties 

of  the  norm  operator 


sM  ^  4-}-(||AC,||2  +  f||A4||2)l 

^  #>-(||AC.||2  +  ||AC'p||2)1  (9) 

Note  that  jjACzlP  +  ||ACp|p  =  HACp  and  hence 
IjACzll  +  IjACpIl  <  •\/2||AC'||,  which  gives  us  the  final 
inequality 


gW  ^  4-} -y2||AC||2l  (10) 

Having  (10)  and  (8)  and  recalling  that  sj  '  and  Xp  are 
both  strictly  positive,  it  should  be  clear  that  if  HACH  is  not 
too  big  then  one  can  find  non-zero  configurations  for  AC 
(or  equivalently  signal  u  and  noise  vector  0  where  and 
Ui  will  both  be  always  positive  which  finally  proves  that  ISS 
of  QP{A,y,  X)  is  not  empty.  ■ 

In  the  following,  we  prove  that  with  a  proper  regularization 
parameter  A,  the  corresponding  signals  in  the  ISS  of  an 
already  recovered  signal  are  guaranteed  to  be  recovered  with 
low  errors  as  well. 

Theorem  4.  Suppose  that  x  =  QP{A,  y  =  Ax  +  e,  A)  has 
the  support  on  p  =  Support{x}  and  Support{x}  C  p 
and  \\x  —  x\\  =  0(||e||).  Let  q  =  Au  +  ^  and  assume 
Support{u}  C  p  and  Ay  =  q  —  y  belongs  to  ISS  of 
QP{A,y,X).  Then  the  error  of  the  estimate  of  u  = 
QP{A,u,  X)  is  in  the  form  of 

14-41  <ii(Ai;"pAp)-i|i(Av^+iiA;^piiiieii) 

Proof.  If  Ay  belongs  to  ISS  of  QP{A,y,  X),  then  by  defi¬ 
nition  of  ISS,  Support{u  =  QP{A,q,X)}  =  p  as  well  and 
therefore 

Up  =  —{Aj'pA:^p)  ^{—Afpq  +  Al) 

where  1  is  an  all  one  vector  with  size  of  Card{p}.  Recalling 
that  q  =  Au  +  ^  =  A.  pUp  -f  ^  and  the  fact  that  A.  p  is  full 
rank,  it  can  be  concluded  that 

Up  =  Up  +  {Ai^pA-^p)  —  Al) 

Since  u  is  supported  on  p 

l4-u||  <  \\A+p\\\\^\\  +  XVm\\iA'^^pA.,^p)-^\\ 


It  should  be  noted  that  the  term  X^/m  in  the  error  bound 
(as  opposed  to  what  it  seems),  is  not  very  large.  Let  us  show 
that  with  an  example.  Without  any  loss  of  generality  assume 
that  we  are  recovering  a  unit-norm  vector  and  as  in  [11], 
the  regularization  parameter  A  equals  to  10ay/\ogn  where 
a  is  the  standard  deviation  of  the  entries  in  the  noise  (here 
the  vectors  e  and  0.  It  should  be  clear  that  since  we  are 
working  in  a  normalized  setting  (i.e.  x  is  unit  norm  and 
the  expected  value  of  the  gram  matrix  of  A  is  the  identity 
matrix),  then  pure  noiseless  samples  Ax  and  Au  should 
have  a  norm  around  one  as  well.  Since  the  dimension  of 
noise  (either  e  or  0  is  to,  we  expect  that  for  a  successful 
recovery,  the  variance  of  the  noise  distribution  (ct^)  to  be 
considerably  smaller  than  1/to,  since  otherwise,  on  average 
the  energy  of  the  noise  is  either  equal  or  larger  than  pure 
samples  (^^(llelli)  =  -^'(IlClli)  =  ^  !)■  Therefore,  in 

this  assumed  normalized  case,  we  expect  that  A  to  be  much 
smaller  than  104(logn)/TO  and  consequently  X\/rn  should 
be  much  smaller  than  lOVlog  n.  Also,  it  is  important  to  note 
that  A  needs  to  be  smaller  than  ||A^j/||oo  since  otherwise  for 
A  >  ||A^y||oo,  X  would  be  fixed  at  zero  which  is  clearly 
wrong  [15]. 

Corollary  5.  If  A  has  weak-RIP  of  order  Card{p},  then 
estimate  of  u  has  the  following  error  for  a  small  positive 
constant  6 

ll'u-ull  <  j^(Av4i-f  Vl  +  (5||0|) 

Proof.  By  the  definition  of  weak-RIP;  \/\  —  5  <  ||A:  p||  < 
\/l  -\-  6.  Plugging  such  value  into  Theorem  4  proves  the 
claim.  ■ 


III.  SIMULATION  RESULTS 

For  our  simulations,  we  could  have  adapted  well-studied 
matrices  with  RIP  (e.g.  Gaussian  ensembles)  as  sensing 
matrices  A  since  they  surely  would  have  isotropy  and  inco¬ 
herence  properties  as  well.  However,  we  have  intentionally 
utilized  SERP  matrices  (introduced  in  [21])  in  the  sensing 
mechanism.  The  reason  for  such  selection  is  that,  SERP  ma¬ 
trices  do  not  have  RIP  of  a  proper  order  with  m  =  Oik  log  n) 
rows.  However,  they  admit  isotropy  and  incoherency  prop¬ 
erties  (although  the  desirable  value  of  /i  =  is  not 

achievable  with  to  =  0(A:logn).  Broadly  speaking,  SERP 
matrices  are  real-valued  sparse  matrices  where  each  column 
is  non-zero  in  d  =  O(logn)  random  row  indices  and  each 
non-zero  entry  is  a  zero  mean  Gaussian  random  variable  with 
the  variance  of  \/d.  We  have  set  d  =  51ogn  for  matrix 
generation  in  our  simulations.  Also  for  solving  LP{A,y) 
and  QP{A,y,  X),  we  have  used  Ll-LS  algorithm  [18]. 

In  practice,  even  when  one  attempts  to  recover  an  exactly 
sparse  vector  x  G  M"  from  compressive  samples  y  G  K™ 
(whether  it’s  noisy  or  noiseless),  LP(A,  y)  and  QP{A,  y,  A) 
virtually  always  return  compressible^  estimates  x.  This  is 
mainly  due  to  finite  hardware  precision,  insufficient  number 

^By  compressible  we  mean  that  most  of  entries  have  small  magnitudes 
instead  of  being  exactly  zero. 
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of  iterations  in  optimization  algorithms  and  similar  related 
issues.  Hence  if  one  wants  to  simulate  our  claims  about  ISS 
then  it  will  be  always  observed  that  solutions  to  the  perturbed 
and  original  optimization  problems  would  be  both  non-zero 
on  all  entries.  Therefore,  ISS  would  trivially  hold  true,  since 
for  instance 

Support{LP{A,y)}  =  Support{LP{A,y  +  Ay)}  =  [n] 

On  the  other  hand,  variations  in  the  support  of  the  solution  to 
an  optimization  problem  might  not  be  abstracted  into  a  single 
number  due  to  its  dimension.  To  address  these  concerns,  we 
needed  to  define  some  notations  and  functions,  presented  in 
here: 

1)  For  a  natural  number  z  and  a  vector  ft,  T{z,u)  is  the 
support  of  top  2  entries  of  ii  with  the  largest  magni¬ 
tudes.  Hence  if  iT[2](.)  denotes  the  hard-thresholding 
operator  which  keeps  z  largest  coefficients  of  its  ar¬ 
gument  then,  T{z,u)  =  Support{H[z-^{u)}.  Clearly 
T{i,  ii)  C  T{i  +  1,  u). 

2)  For  a  natural  number  z  and  two  vectors  x  and  u,  define 

f{z,u,x)  =  min  6 
s.t.  T{z,u)  C  T{b,x)  (11) 

In  other  words,  all  top  z  largest  entries  of  u,  can 
be  found  in  top  f{z,u,x)  largest  entries  of  x  and 
furthermore,  f{z,  u,  x)  is  the  minimal  value  with  such 
property. 

It  can  be  easily  verified  that  if  magnitudes  of  two  vectors  u 
and  X  have  exactly  the  same  pattern,  then  f{z,  u,  x)  should 
be  equal  to  z.  Note  that,  this  condition  is  much  stronger 
than  just  to  say  that  top  1%  coefficients  of  u  and  x  have 
the  same  support  for  some  value  of  1.  For  instance  assume 
that  X  =  [3  2  1  0]^.  Then  top  three  largest  coefficients  of 
u  =  [2  1  3  0]^  and  X  have  the  same  support.  However, 
stating  that  f{z,u,x)  =  z,  implies  that  |mi|  >  \u2\  >  lual 
(i.e.  the  same  magnitude  pattern  in  x). 

Recall  that  in  this  paper,  m  is  the  number  of  rows  in  A, 
n  denotes  the  length  of  the  unknown  vector  x  (and  clearly 
the  number  of  columns  of  A),  k  is  the  number  of  non-zero 
entries  of  x,  the  noise  variance  is  shown  by  cr^  and  the 
regularization  parameter  of  Lasso  is  A.  Given  a  fixed  set  of 
parameters,  we  run  the  following  algorithm:  we  generate  a 
random  SERF  matrix  A,  a  unit  norm,  non-negative  vector  x 
with  a  random  support  and  magnitudes  and  a  Gaussian  noise 
vector  e.  Depending  on  the  noise  variance,  we  estimate  it  by 
X  =  LP{A,y)  or  X  =  QP{A,y,  X)  for  y  =  Ax  +  e.  To 
generate  the  perturbed  signal  u,  we  randomly  change  x  in 
a  indices  among  support  of  x  and  make  it  non-zero  in  /3 
extra  random  indices  among  T{2k,  x)/ Support{x}.  Hence, 
Support{x}  C  Support{u}  and  Card{Support{u}}  = 
Card{Support{x}}  +  /3.  After  this  step,  we  normalize  u 
and  make  sure  that  u  would  remain  non-negative  as  well. 
In  our  simulation,  there  will  be  no  perturbations  in  noise 
(Ae  =  ^  —  e  =  0)  to  limit  enormous  degrees  of  freedom 
in  simulations.  To  have  an  idea  on  how  similar  or  distinct 
are  vectors  x  and  u  in  our  simulation,  we  report  the  Signal 


to  Noise  Ratio  (SNR)  between  x  and  u  =  x  +  Ax  in  the 
following.  Let  ii  be  the  estimate  of  a  compressed  sensing 
framework  for  u  from  samples  q  =  Au  +  e.  If  ISS  holds  in 
that  framework,  then  f(z,u,x)  =  z. 

We  have  simulated  each  of  the  following  three  configura¬ 
tions  for  one  thousand  times.  No  attempt  has  been  made  to 
optimize  the  regularization  parameter  A  which  was  fixed  at 
A  =  0.002: 

SIMl:  TO  =  240,  fc  =  40,  n  =  1000,  a  =  fc,  /3  =  0  and 
cr  =  0.  In  this  configuration,  there  is  no  noise  in  samples. 
Also,  X  and  u  are  two  independent  unit-norm  signals  with 
the  same  support.  However,  at  each  of  1000  runs,  both  the 
support  and  also  the  matrix  A  would  change  randomly.  The 
average  and  median  SNR  between  unit  norm  vectors  of  u 
and  X  had  been  only  3.11  and  3.05  dB  respectively.  Hence 
although  both  signals  had  the  same  support,  they  were  totally 
different  signals  in  simulations.  The  correlation  coefficient 
between  SNR  of  u  to  u  and  SNR  of  x  to  x  was  computed 
at  0.986.  This  means  that,  whenever  x  was  recovered  with 
smaller  errors,  then  the  estimate  of  u  would  have  smaller 
errors  as  well.  This  meaningfully  high  correlation  when  SNR 
of  X  to  u  is  that  low,  acknowledges  our  claim  that  the  quality 
of  the  recovery  process  in  the  noiseless  and  RIPless  case  is 
independent  of  the  signal  value  and  only  depends  on  the 
support  of  signals. 

SIM2:  TO  =  600,  k  =  100,  n  =  5000,  a  =  /3  =  2  and 
=  0.001.  The  average  and  median  SNR  for  sampling 
process  in  all  2000  recoveries"^  both  were  approximately 
32.30  dB.  The  average  and  median  SNR  between  x  and  the 
perturbed  vector  u  were  respectively  19.01  and  18.07  dB. 
The  correlation  coefficient  between  SNR  of  recoveries  of  u 
and  X  was  0.995. 

SIM3:  TO  =  1200,  k  =  200,  n  =  3000,  a  =  10,  /3  =  3 
and  cr^  =  0.001.  The  average  and  median  SNR  for  sampling 
process  in  all  2000  recoveries  both  were  approximately 
29.38  dB.  The  average  and  the  median  SNR  between  x  and 
the  perturbed  vector  u  were  13.42  and  13.18  dB  respectively. 
Hence,  perturbed  vectors  u  are  distinctly  different  from  x 
in  this  scenario  as  well.  The  correlation  coefficient  between 
SNR  of  recoveries  of  u  and  x  was  0.996. 

Left  column  of  Figure  1  shows  how  similar  signals  x 
and  u  were  in  each  iteration.  Qualities  of  recoveries  are 
presented  in  the  middle  column  of  Figure  1 .  The  right  column 
of  the  same  figure  investigates  whether  ISS  holds  for  these 
simulations  by  utilizing  the  function  of  /(z,  u,  x),  introduced 
in  equation  (11).  As  stated  before,  ideally  we  expect  that 
/(z,  u,  x)  =  z.  However,  it  should  be  noted  that,  the  function 
f{z,u,x)  is  sensitive  to  small  permutations  in  supports  of 
it  and  x.  For  instance,  assume  that  three  largest  coefficients 
of  ft  and  X  are  at  indices  of  a,  b,  c  nd  6,  c,  a  respectively. 
Then  although  these  two  vectors  have  the  same  support, 
we  would  have  /(1,m,  x)  =  f{2,ii,x)  =  f{3,u,x)  =  3 
which  is  away  from  f{z,  it,  x)  =  z.  This  should  explain  some 
straight,  horizontal  lines  in  our  plot.  Also,  since  in  SIM2  and 

‘*In  each  of  one  thousand  iterations,  one  set  of  samples  is  measured  for 
u  and  another  set  of  samples  is  measured  for  x. 
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Fig.  1:  Top  row:  SIMl,  middle  row:  SIM2  and  bottom  row  is  SIM3.  Left  column:  SNR  (in  dB)  between  the  original  signal 
X  and  the  perturbed  signal  of  u  =  a;  +  Aa;  as  a  function  of  the  simulation  number.  Middle  column:  SNR  of  recoveries  for 
u  and  a:  as  a  function  of  the  simulation  number.  Right  column:  f{z,u,x)  as  a  function  of  z.  If  the  magnitude  of  the  plot 
at  row  i  and  column  j  is  t,  it  means  that  out  of  1000  simulations,  in  t  times  f{j,u,x)  =  i. 


SIM3,  Card{Support{x}}  <  Card{Support{u}}  then  we 
expect  that  f{z,u,x)  to  be  fixed  at  Card{Support{u]}  for 
small  values  of  z,  which  explains  the  bright  horizontal  lines 
for  those  values  of  z.  As  a  final  note,  our  arguments  in 
the  previous  section  were  for  the  worst  case  scenario  and 
in  practice  there  are  many  perturbed  problems  which  are 
still  in  the  ISS  while  they  do  not  meet  equations  (8)  and 
(10).  On  the  other  hand,  we  had  generated  perturbations  of 
Aa;  =  u  —  x  randomly  in  our  simulations.  Consequently,  true 
ISS  conditions  might  be  violated  in  our  simulations,  leading 
to  anomalies  in  plots  f(z,u,x).  However,  as  it  is  clearly 
illustrated  in  Figure  1,  the  notion  of  ISS  for  compressed 
sensing  holds  to  very  good  extent  even  in  those  random 
simulations  since  the  function  f{z,u,x)  looks  like  to  be 
equal  to  z  in  most  cases 


IV.  CONCLUSION 

Invariant  Support  Set  (ISS)  of  an  optimization  problem 
is  the  set  of  perturbations  in  the  objective  function  or 
constraints  of  that  problem,  such  that  the  support  of  the 
solution  stays  intact.  In  this  paper,  we  showed  that  in  a 
RIPless  scenario  for  the  compressed  sensing  framework,  any 
problem  which  can  be  solved  correctly,  would  have  a  non¬ 
empty  ISS.  This  means  that  in  the  noiseless  case,  successful 
recovery  of  a  fixed  but  arbitrary  signal  guarantees  that  all 
other  signals  with  the  same  support  and  sign  pattern  could 
be  recovered  exactly  as  well.  In  a  noisy  setting  however, 
good  recovery  of  a  fixed  but  arbitrary  signal  guarantees 
that  there  exist  infinitely  many  other  signals,  (possibly  even 
with  different  supports),  which  can  be  recovered  with  good 
precision  from  their  noisy  compressive  samples. 
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